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EDITORIAL 


People live with all kinds of digital objects. These range from social networks 
to cutting-edge connected objects. They are now a fully-fledged part of our 
aesthetic relationship with the world, on a par with architecture or decorative art. 
A certain widely distributed and highly standardised aesthetic of digital media has 
been created, producing strong brands in users’ minds. Users are barely aware of 
this aesthetic, which is well thought out. Conditioning through design pre-empts 
everything that the individual handles or sees in the digital world. 


In the wake of Bauhaus, where design is based on the search for functional 
aesthetics, digital technology has become a relevant field of application in 
problem-solving. For, beyond the desire to avoid ugliness that “does not sell’, to 
quote Raymond Loewy, the digital promise is so broad that the functions provided 
by design seem endless, giving the key players that own them the expectation of 
a substantial return on investment. 
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Web giants have understood this by competing to attract users, customising 
their experience, influencing — as subtly and as substantially as possible — their 
behaviour, in all fields, from leisure to politics. They stand as a compass to navi- 
gate the abundance of content and the anxiety of the non-optimisation of time. 


But this model is no longer followed. Maybe because it seems to “take advantage” of individuals, their “malleability” 
and tendency to become accustomed to simplicity. Concerns regarding personal data are a strong signal of the 
discomfort felt by users themselves. 


In this regard, the General Data Protection Regulation (GDPR) is a major response and a first essential milestone 
towards greater transparency and a reaction to the crisis of confidence. It offers regulators several legal and 
technical instruments to make users the focal point of the data economy. 


But, to fight the model of individuals who are “objects of technology’, design can act as a rampart and deploy 
its firepower. 

The aim is not only to make objects beautiful but to propose an aesthetic to support another digital project. 
This project is not simply functional but humanistic, targeting sustainable goals that users can master; a project 
more imbued with common sense regarding the real needs of users taken in all their complexity. 

Prosaically, the aim is not to be deceived, to fully consent to the effort that companies want to relieve us of and 
ultimately say “yes” together. To do that, in practical terms, the interface is far from cosmetic. 

Design then reaches its full meaning, that of aesthetics to support humanity. It is beautiful as it is rooted in that 
humanity. 


This publication therefore aims to project some ideas to build this digital aesthetic. It addresses the entire digital 
ecosystem by giving some operational recommendations to strengthen the control and choice to which users are 
entitled. The CNIL intends to participate in this and considers the attention taken with design solutions as a poten- 
tial framework on which to build its legal and technical expertise and better fulfil its mission to protect freedoms. 


May this document allow readers to be more reactive to digital tools, beyond an exclusively instinctual reaction. 
May this document lead to a virtuous circle restoring the validity of usage, in everyone’s interests. May this 
document convince that the beauty of objects or forms is nothing unless we look 

at them with our eyes wide open. 


In the industrial age, design had established that technical progress should support Isabelle 

everyone, through the serial production of objects useful to individuals’ everyday Falque-Pierrotin 
lives. It now needs to help build a “digital aesthetic” for all, allowing individuals to Président of the CNIL 
find their rightful place. 
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“But where there is the peril, 
also grows the saving power.” 
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WHAT LINKS BETWEEN INTERACTION DESIGN, INTERFACES AND DATA PROTECTION? 


What links between interaction design, 
interfaces and data protection? 
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For better and for worse, digital tools have become our 
everyday assistants: ordering food, determining a route 
or even finding a soul mate, they propose, in the words of 
Dominique Cardon, the possibility of “relieving humans of the 
more mechanical aspects of their activities, freeing them for 
higher, more complex or more ambitious cognitive tasks!”. 


Facilitating the work of humans is only ever the definition of 
a tool: a means to an end that would be difficult or impos- 
sible to attain for an individual. Why in the digital domain, 
would this function have such revolutionary consequences? 
What's so special about these objects to bring out issues 
that differ so greatly from those raised by a hammer, a car 
or a pair of glasses? 


Technology has never been neutral. It is “a kind of rela- 
tionship with the world”, as Heidegger wrote. He believed 
its essence lay in the unveiling: technology brings a hidden 
potential of the world for the individual to grasp”. In other 
words, technology corresponds to a reconfiguration of pos- 
sible outcomes, born of the encounter between the creative 
work of an object and its hand-over to individuals. The design 
of tools is not a trivial process, inert in terms of users and 
society. Tools shape us as much as we shape them. 


1 Dominique Cardon, À quoi rêvent les algorithmes, Seuil, 2015 


2 Martin Heidegger, La question de la techniquel, 1954 
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José Alejandro Cuffia on Unsplash 


The advent of digital technology especially changes the 
scale at which this shaping is liable to take place: tools can 
be disseminated very broadly and quickly, furthermore they 
can finely (and discreetly) adapt to their users. Thus, they 
have all the characteristics to be able to transform society 
in depth. 


Our goal is to identify key areas in digital tools and services 
where these influence dynamics are focused. The design of 
human-machine interfaces, far from being a simple matter of 
ergonomics, comes with crucial issues in terms of the ability 
to act and configure possibilities of choice, so it is essential 
to focus on understanding to better control them.* 


FROM INTERFACES TO 
INTERACTIONS, WHAT ARE 
WE TALKING ABOUT? 


Almost everything is an interface. The skin allows us to react 
to our environment. A pen can visually express an idea from 
hand to paper. A smartphone screen lets you browse and 
modify digital reality. Although almost invisible because they 
are rooted in our everyday lives, these interfaces are essen- 
tial to our perception of the world and our ability to act in it. 
In the broadest sense, they can be defined as common areas 
with different entities, systems or sets, including physical or 
sensitive characteristics allowing them to share and interact 
through common representation modes. 


In the digital world, interactions between reality and virtuality 
are mediated by human-machine interfaces (HMI). These 
are the fruit of the joint work of engineering (which defines 
its action and reaction capacity) and design (which deter- 
mines the representations - visual, architectural, verbal, etc. 
- guiding users in their interactions with machines). 


The ability to implement these interactions effectively is of 
prime importance for the use of these interfaces and the 
systems that underpin them. The cognitive psychologist, Don 
Norman, has stressed the importance of the “discoverability” 
of technical objects: it is essential for potential users to be 
able to easily determine the actions they can perform with 
them - for example, instinctively knowing which way a door 
opens“. This principle is based on two key levers: 


- indicating to the user interactions that are possible with 
the interface, through affordance, i.e. all of the potential inte- 
ractions between an interface and the agents in relation to 
it; signifiers, i.e. indicators showing how to operate the inter- 
face; and constraints, i.e., the limitations of possible actions. 


- allowing the user to conceptually represent the system 
by revealing the logical links between an action and its effect 
on the system through the mapping of these links and feed- 
backs informing the result of the action taken. 


3 Agency: ability to act independently and make their own choices 4 Donald Norman, 
The Design of Everyday Things, 1968 


4 Donald Norman, The design of Everyday Things, 1968 
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Once these principles established, HMIs do not act alone, 
but jointly within interface ecosystems, which can, for 
example, mix virtual (graphic interface) and physical 
(smartphone). Thus, a virtual interface specific to a digital 
service is always constrained by the affordances of the 
equipment on which it is instantiated. 


Take the example of an instant messaging application for 
smartphones. The interface specific to the service allows the 
user to communicate with its contacts. This application provi- 
des a set of features whose interactions are determined by 
the affordance of the interface (e.g, tapping with the finger 
on certain elements), constraints (such as the hierarchy of 
content accessibility), and finally its use is guided by signi- 
fiers (such as icons or colours). All this has to intervene in 
the constrained hardware space of the smartphone (the size 
of its screen, the fact that it is touch sensitive...). 


These paradigms are expressed as the different comple- 
mentary practices of the design professions. Thus, the fact 
of telling the user what they can and cannot do falls within 
the user interface design (or UI) which strives to build a 
coherent visual language. It is built through the design of 
interactions or IxD, i.e. how the interface interacts between 
the system and the user, to enable it to achieve its objec- 
tives. The concept of user experience (or UX) has recently 
emerged; it includes an expanded version of the design of 
interaction focusing on the user journey as a whole, focusing 
on the emotional quality of experience and the commitment 
between a service and its users. 


With this toolbox and through their ability to coexist and 
factor in the affordances of these different interfaces, 
designers and service developers design tools and digital 
pathways of users in search of the ideal interface. 
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UBIQUITOUS, CUSTOMISED, SEAMLESS: 


THE DEFAULT INTERFACE 


The quality of the user experience has become the magic 
word in the design of any digital service or product. 
This underlying ideal in the development of interfaces 
is embodied in a set of principles and tools that aim to 
improve the experience proposed by combining simplicity, 
customisation and multi-modality. 


This quest begins with the search 

for simplicity: “less is more” pro- “u 
posed by the architect Ludwig 
Mies van der Rohe, taken over by 
the concept of “design is as little 
design as possible” of Dieter Rams, 
and theorised by John Maeda in 
his laws of simplicity are still refe- 
rences. In HMI, this results in a race 
to smooth, seamless and friction- 
less user experiences to achieve 
greater efficiency that is conside- 
red preferable by many stakehol- 
ders in the sector. Simplicity is seen 
today as a way of supporting the 
principle of efficiency, the dominant paradigm of our society, 
so that users do not waste time and quickly perform what 
they want through the interface. This axiom has become an 
iron law of web design, formalised by Steve Krug in his book, 
Don't make me thinkë: “I should be able to understand what it 
is and how to use it without straining to think about it”. These 
laws, axioms or principles are often considered indisputable 
and permeate through the best practices of professionals. 


A second guarantee of a quality experience is the custo- 
misation of services to the user, by a user-centric design 
and by the algorithm. This is supposed to define the real 
needs of users by conducting a set of searches on them 
and their environment, to understand their problems, what 
irritates them, their way of acting and thinking in order to 
define the principles of uses and interactions of the service. 
The processing of users’ personal data aims to antici- 
pate their needs, to show one thing rather than another. 
The customisation trend does not prevent an increasing 
standardisation of experiences and user pathways: instead, it 
supports it. The multiplication of systems design is an obvious 
symptom. The interfaces are increasingly standardised in 
terms of their elements, their structures and their behaviours. 
No wonder: the customisation of the interface will be more 


The customisation of the 
interface will be more effective 
if it is based on user codes 
distributed across all services 
and already established among 
users in a sort of digital interfaces’ 
grammar, a lingua franca of user 
experiences. j 


effective if it is based on user codes distributed across all ser- 
vices and already established among users in a sort of digital 
interfaces’ grammar, a lingua franca of user experiences. 


Finally, today’s experience has to be multimodal. With 
recent developments in ubiquitous 
digital technology, a service can 
no longer settle for being acces- 
sible through a single medium; it 
must multiply them while ensuring 
continuity and consistency across 
experiences. Smart voice assistan- 
ces are an example: named Alexa, 
Siri, Cortana or Google Assistant, 
these assistants are available on 
our smartphones, our compu- 
ters, connected speakers or our 
vehicles® and gradually take over 
all the objects in our environment. 
The proliferation of media allows 
the emergence of new means of 
interaction. Called natural user 
interfaces (NUI), these new modes of interaction claim to 
be revolutionary by a supposed spontaneous understanding 
and the gradual learning curve they offer their users. Besides 
fluid interactions, these interfaces also open the door to 
experiences that increasingly focus on emotions, as we will 
explore in a series of articles on LINC.cnil-fr. 


5 Steve Krug, Don't make me think, 2005. 


6 https://www.lesnumeriques.com/voiture/ouvrir-sa-volkswagen-avec-siri- 
est-desormais-possible-n80253.html 
(viewed on 20/12/2018) 
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WHY IS DESIGN CRUCIAL 
FOR PRIVACY? 


In 2009, Ann Cavoukian, the Information and Privacy 
Commissioner of Ontario (Canada), proposed the formula 
of “privacy by design”, a concept taken up by academic and 
professional productions, but rather as a general philosophy 
of respect for privacy - through seven key concepts? - than 
as a specific stance aimed at professionals designing inter- 
faces or interactions. Here it is more of a 
complementary way to tackle the issue of 

data protection for the usual interlocutors: 

lawyers and engineers. “u 


The interface is 
the first object of 
mediation between 
the law, rights and 
individuals. ” 


The General Data Protection Regulation 
(GDPR) inserts in Article 25 the concept of 
“privacy by design and by default”. It stresses 
the need to implement appropriate protec- 
tive measures according to the context of 
processing and the privacy risks of data 
subjects. In parallel, the article emphasises 
default respect for privacy by emphasizing 
the importance of the active participation of data subjects in 
protecting their privacy by asking them to intentionally acti- 
vate invasive features, such as sharing data with third parties. 


While Article 25 does not explicitly seem to address desi- 
gners, it allows us to look at and highlight “privacy design”, 
how different design techniques are used in the staging 
of services for - and sometimes at the expense of - the 
protection of individual data, especially with regard to the 
major principles of transparency, consent and rights of indi- 
viduals. It acts as an opening to the association of design 
and regulation. 


Interface design did not wait for the General Data Protection 
Regulation (GDPR) to influence our lives, and sellers did not 
wait for digital technology to try to guide our actions and 
persuade us to purchase their products. We have long been 
influenced in our movements and actions by architectures of 
choice designed and implemented by others. For example, 
the mass retail market has long modelled its hypermarkets 
so that the customer pathway is guided by colour codes 
or pre-established paths to maximise purchasing, from the 
location of water packs to sweets placed at the checkout 
counter. Think for a moment about the pathway taken by 
visitors in an Ikea store, for example... 


WHAT LINKS BETWEEN INTERACTION DESIGN, INTERFACES AND DATA PROTECTION? 


Yet these questions take an unusual turn when they are 
applied to interfaces and digital services that use and abuse 
deceptive design methods to hook us and better collect and 
process our data for goals that we do not always control. The 
design of these services affects us all because we depend 
on choices made by those building them, of what is repre- 
sented, and by extension also what is not 
(James Bridle). If these issues relate to the 
contexts in which the data concerning us is 
processed and used, UI design and the way 
such interfaces allow us to make conscious 
decisions becomes a central point. The inter- 
face is the first object of mediation between 
the law, rights and individuals. 


But the techniques of playing with our atten- 
tion and our cognitive biases to develop 
manipulative and/or misleading interfaces 
(see below) have a direct impact on our abi- 
lity to uphold our rights. We are so influenced and trained to 
share more and more, without always being aware, ultimately, 
that we are jeopardising our rights and freedoms. We there- 
fore need to explore how design is used in the production 
of digital services to understand its positive and negative 
uses for all of us. 


The aim of this document is thus to put the design of inter- 
faces at the centre of the regulator's concerns, as it is 
already at the centre of relations between individuals and 
service providers. 


7 Ann Cavoukian, Privacy by Design : The 7 Foundational 
Principles, Information and Privacy Commissioner of Ontario, 2009. 
https://www.ipc.on.ca/wp-content/uploads/ 

Resources/7 foundationalprinciples.pdf 

(viewed on 07/12/2018) 


8 Proactivity, the default protection, protection by construction, 
favouring a positive analysis and not zero reasoning, end-to-end 
protection on the whole data life cycle, visibility and transparency 
and, finally, respect for informational self-determination. 


9 James Bridle, New Dark Age : 
Technology and the End of the Future, 2018 


Relations, 
individuals and 
services: 

“It's Complicated” 


“The Critical Engineer recognises that each 
work of engineering engineers its user, 
proportional to that user s 
dependency upon it”. 


The Critical Engineering Manifesto 
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Relations, individuals and services: 
“It's Complicated” 


njaaa iai D Dr ea 
FOR OUR ATTENTION 


Digital technology has this in common with the mass media: 
LEAD TO THE its dominant economic model is essentially advertising. 

Attracting people’s attention to target them with advertising 
MAN | PU LATIO N is a central commercial issue of major digital players, in par- 

ticular big platforms operating on two-sided markets, where 
OF USERS? consumers are less customers than products. 


Patrick Le Lay, then CEO of TF1 (leading French 
TV-Channel), thus defined his business in 2004: 
“What we sell to Coca-Cola is available brain time”. 


E 
FA 


Pexel, CCO, Kaique Rocha 


Nathan Jurgenson, a sociologist who works, among others, 
for Snap (parent company of Snapchat), recalls that “the 
original sin at the birth of the mass media was to tie profit 
to the quantification of attention”? and this sin largely pre- 
dates the web and its permanent measurement of attention. 


The first deviation of such an economic attentional mecha- 
nism is based on his assumption that “figures measure 
behaviour: someone says something, if it’s interesting, the 
figures will show, and vice versa. [...] People are starting to 
try to influence these figures, measurement then goes from 
being a means to an end. Whether on Twitter, Instagram 
with hearts, or for the number of clicks on a page [...] the 
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metrics become the conversation. Popularity is what is 
interesting”. 


The attention economy has thus become inseparable from 
economic data as the raw material for all the success indi- 
cators. The consequences for individuals are therefore very 
different from the days when television reigned. Whereas 
traditional media were based on fragmentary samples (e.g. 
in France, the Médiamétrie panel which had a TV ratings 
boxes), current digital and online media now promise their 
advertisers increasingly fine segmentation marketing, based 
on the real-time collection and analysis of individuals’ activity. 


Focus on... 


On 27 October 1994, Wired magazine applied in its 
online version a first, admittedly crude, but certainly 
effective, attempt at deceptive design (dark pattern) 
aimed at its readers. A banner, an object previously 
unknown to users of the website, was visible on 
the top of the page with this inscription: “Have 
you ever clicked your mouse right here? You will”. 
Since then, as James Williams described", 
computer engineers, designers and statisticians 
spend their days thinking about how to direct 
attention and user behaviours towards goals that 
were not originally theirs. As Jeff Hammerbacher, 
formerly of Facebook, says: “the best minds of my 
generation are thinking about how to make people 
click ads, and that really sucks”. 


10 Annabelle Laurent, “ Sur les réseaux sociaux, le contenu n'existe que pour maximiser les likes ', 
Usbek & Rica, https://usbeketrica.com/article/reseaux-sociaux-contenu-existe-maximiser- 
chiffres-nathan-jurgenson (viewed on 18/12/2018) 


11 James Williams, Stand Out of Our Light, Freedom and Resistance in the, Attention Economy, 
University of Oxford, 2018 
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Because these techniques are 
combined with a constant search 
to capture attention, platforms are 
encouraged to act on their psycholo- 
gical mechanisms in order to benefit 
from them. 


Exploring the economy of attention 
through the data prism therefore 
amounts to questioning the conse- 
quence of this race to capture atten- 
tion. Creators of services only show 
the positive effects: the consumer 
would ultimately be practically rewar- 
ded for viewing content which must 
be ever more interesting to obtain 
their approval. The advertising to 
which consumers is exposed would 
be so relevant that they would cor- 
respond to services provided and not 
be an annoyance. 


This analysis however ignores what behavioural economics 
and analysis tell us about the strategies and practices of 
economic players. Platforms do not only seek to capture 
the attention, but sometimes, in this way, they divert from 
the underlying economic model … As noted by sociolo- 
gist Dominique Boullier, “Marketing knows how expensive 
it is to gain customers and how important it is, therefore, 
to keep those we already have. For that, it is necessary to 
fight attention-hopping, against this permanent infidelity 


FOCUS ON... 


LINC, exploration partner 
“For retro-design 
of attention” of the FING 


LINC is a partner of the exploratory pro- 
ject “for a retro design of attention”'?, 
launched in January 2018. This project 
aims to analyse (or study) how our at- 
tention is captured by interfaces and to 
suggest new avenues for responsible 
attention. The results will soon be publi- 
shed on the Fing website. Throughout the 
project, summary articles are published 
on Internetactu.net (#attentionbydesign). 


encouraged by the alert policy and 
hat the same marketing services 
and media implement to attack 
competitors’ customers”. He comes 
o the following conclusion: “the 
whole point of this struggle to cap- 
ure the available brain time is to 
reduce to the extreme any hesita- 
ion and conscious arbitrations, to 
create a form of naturalness which 
poses no problem, which will seem 
very economic at the cognitive level. 
Capturing attention becomes the 
ultimate form of loyalty, which will 
protect the customer from aggres- 
sion by competing attention sensors 
in an immunity bubble. 


The aim is therefore not only to cap- 
ture attention innocently by being 
the best and providing the most 
interesting or useful content. It is a 


vicious struggle to control attention and its economic, social 
and cognitive mechanisms. The methods used by those buil- 
ding content - nudge, dark patterns or deceptive design 
- which we present in the following section, not only act on 
the attention of individuals but also on their behaviour and 
agency. These effects have a direct link with the protection 
of rights of individuals and their personal data, as they may 
have to share more and more without being necessarily 
conscious about it. 


12 http://fing.org/?Pour-un-retrodesign-de-l-attention&lang=fr 
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Focus on... 


Our hypervigilance as the first 
breach of attention 


Interviewed by InternetActu, the neuroscience 
researcher and clinical psychologist, Albert 
Moukheiber repositions our reactions to demands 
for attention over a longer period, linked to the very 
nature of the human being: “Our hunter-gatherer 
ancestors had to deal with dilemmas. When they 
heard a noise in the forest, they had to judge if it 
was the wind or a predator. There was a benefit in 
making one choice rather than the other. Missing an 
alert was costlier than raising the alert for nothing. 
We developed hypervigilance ... preferring to make 
a mistake rather than die. And this hypervigilance 
reflex has stayed with us”. Humans instinctively 
make the choice of processing information quickly 
and responding to visual and sound signals, like 


Cognitive biases at 
the centre of attention 


Taking advantage of all cognitive biases, those faults in our 
rationality that hinder our free decision-making capacity, is 
one of the fundamental levers in the race to capture the 
attention of Internet users. Various work, including that 
of psychologists Daniel Kahneman and Amos Tversky in 
the 1970s, questions the model developed in the 1920s 
by Edward Bernay’? of homo oeconomicus: an individual 
acting perfectly rationally according to their interests and 
objectives and on which classic economic theory was based. 
From various experiments, they show that our perceptions 
and behaviours are largely guided by our physical, social 
and cognitive development and, consequently, we mostly 
take apparently irrational decisions. Cognitive biases, those 
mental structures that limit our rationality, have subsequently 
been identified in many areas. 


In the digital world, research has led to the questioning of 
the free and informed nature of choices made by individuals, 


smartphone notifications. Our attentional hypervigi- 
lance explains why we are startled at an unforeseen 
event or why we continue to focus our attention 
when faced with weak signals. This is the founda- 
tion of the error management theory updated by 
David Buss and Marie Haselton, stressing that it is 
costly to integrate all the details of our environment 
while remaining fast and accurate, which explains 
why we prefer to take the less costly mistake. For 
these reasons, we always tend to give into the call 
of the platform when it emits a signal, warning us 
of a change, of a novelty; which by innate reflex, 
we do not want to miss. This the platforms have 
understood. 


particularly in terms of sharing personal data. For many, 
like Alessandro Acquisti', our cognitive biases explain the 
famous privacy paradox, according to which we share huge 
quantities of personal information online while worrying 
about the consequences of this sharing. We therefore decide 
to share personal data without taking into account all the 
elements of context and without being fully aware of the 
implications of this approach. 


When we look at the individual’s capacity for action and 
agency, the illusion of control is a cognitive bias with parti- 
cularly powerful effects. Considerable work has isolated this 
tendency of individuals to grasp elements that give them the 
impression of controlling a result that nevertheless they do 
not initiate. Thus, a famous study showed that it was much 
easier to get people to donate by adding the phrase “but you 
are free to accept or refuse.’ According to the researchers 
behind this work (especially Nicolas Gueguen and Alexandre 
Pascual), the mere mention of freedom is enough to disarm 
the mistrust of any threat to our freedom (reactance, accor- 
ding to the term used in psychology). 


13 Edward Bernays, Propaganda, 1928. 


14 Alessandro Acquisti et al, Nudges for privacy and security: understanding and assisting users" 
choices online. ACM Computing Surveys, vol. 50, n°3. 2017 
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Focus on... 


Cognitive biases in action: 
practical case 


When a person reaches the middle of an article 
on the internet, they see a banner appear asking 
them to log into the site to continue to read. The 
required fields (surname, first name, date of birth, 
email address) are directly accessible. Next to a 
check box, a link leads them to the terms of use and 
privacy policy, two dense documents of ten pages 
each. The “continue reading” button appears in blue 
at the bottom right, while a “leave the site” link is 
featured in grey, giving the possibility of abandoning 
this activity. 


Annoyed by the untimely interruption and interested 
by the article in question, the person hurriedly fills in 
the fields, opens the link to the attached documents, 
scans them rapidly, checks the “accept” box and 
returns to reading the article. 


Several cognitive biases have come into play in this 
short sequence. 


e Anchoring effect: the user has already seen this 
type of banner dozens of times and is used to dea- 
ling with it without paying much attention. 


e Loss aversion: we tend to value a benefit 
already in our possession more than an object 
we could acquire. Here, the fact that the person 
in question has already started to read the article 
without knowing that it was going to be withdrawn 
increases their commitment to reading it. If the login 
had been requested beforehand, they would cer- 
tainly have been less inclined to provide personal 
information in this case. 


+ Information overload: in the presence of an 
excessive amount of information, we tend to ignore 
it in totality rather than selecting the relevant ele- 
ments. Here, the fact that the conditions of use and 
privacy policy are long and complex decreases the 
probability that they will actually be read and consi- 
dered. Although the person has made the effort to 
open these documents when they appear, they 
decide not to factor them into their choice to log 
into the site. 


e Framing effect: the way things are presented 
influences our decisions. Here, the fact that the 
option leading to the creation of an account is indi- 
cated in a visually attractive way (in blue) and with 
an incentive vocabulary (“continue reading”), while 
the possibility of leaving the site is visually and ver- 
bally less attractive, aims to influence the actions 
of the individual. 


¢ Hyperbolic updating effect: we tend to pay 
more importance to immediate events than to those 
that could occur in the future. The immediate bene- 
fit of access to the article therefore prevails over 
future, possibly disturbing, consequences of the 
sharing of personal data with the relevant site. 


¢ Optimism bias: in the same vein, the user tends 
to underestimate the probability that the fact of 
having logging into the site could have negative 
consequences for them. 
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HOW DO DIGITAL SERVICES HOOK US? 


The weaponization of our habits 


Some experts are quick to tout the capacity of digital tech- 
nology to transform habits into weapons, according to the 
expression used by Nir Eyal, the author of Hooked: how 
do successful companies create products people can't 
put down. This book is a practical and somewhat reflexive 
guide to ethical and societal issues aimed at entrepreneurs 
wishing to develop products that create habits, which we 
use without thinking. It is possible, in his opinion, to develop 
a virtuous circle (from the perspective of the entrepreneur 
portfolio) aiming at the natural use, without external stimuli, 
of a product or service by its user. His hook model goes 
through four phases that must be repeated: trigger, action, 
reward and investment in the service (time, money, content, 
equity, effort or.. data). 


This model is based on a number of strengths: 


e The repetition of cycles eliminates the need for an exter- 
nal trigger to lead to an internal trigger: you start Instagram 
or Facebook even without notification. 


e Variability of the reward is very clearly based on our 
“playful” brain. Feedback loops exist all around us in the 
design of objects: when you press the switch of a lamp, 
you expect it to light up. That in itself does not make us to 
press the button more often. However, if we get a variable 
reward (a new colour for lighting, a sound bonus or other 
surprise action), the service will be able to create an appe- 
tite, a desire... 


e User habits become an asset to the company and give it 
a competitive edge. By increasing loyalty, companies gain 
a monopoly over the mind. 


In an article, published in The Atlantic’? in 2012, the video 
game designer and author lan Bogost recalls how the 
first generation of smartphones from Research In Motion 
(Blackberry) had helped to change the behaviour of users, 
by hooking them to a little red LED that flashed when a 
message had been received, well before Apple or WhatsApp. 
The author recalls that RIM products launched a chain reac- 
tion that changed our social behaviour in ways we still do 


not completely understand. Maybe in 50 years’ time, current 
response reflexes to permanent calls for attention from our 
digital companions will seem as harmful and ill-considered 
as the relationship our elders had in the inter-war years with 
the social role of smoking... 


Playing with our emotions 


The ultimate goal of the entrepreneur, in Nir Eyal's opinion, 
would be to move from external triggers (notifications, email, 
etc.) to internal triggers, which cannot be seen, heard or 
touched, but rely our cognitive biases, our psychological 
needs and emotions. 


The best-known example of these endogenous triggers in 
the world of social networking services is the famous FOMO, 
or Fear Of Missing Out. It is the fear of missing something 
important, useful or entertaining. It is this fear that drives 
us to check our phone dozens of times to continue the 
never-ending scrolling of Instagram or Pinterest images, or 
to check our emails or look at the Twitter feed. Fear, which 
in this case is created by the tool, is the salve that relieves 
irritation of which it is the cause... 


This example is far from trivial. As Nir Eyal says, “emotions, 
especially negative ones, are potent endogenous triggers 
and strongly influence our daily routines. Feelings of bore- 
dom, loneliness, frustration, confusion and indecision often 
induce mild pain or irritation and provoke an almost instant 
reflex to mitigate this negative feeling”. 


It is often said that the objective of designers of products, 
and more generally of digital entrepreneurs, would be to 
eliminate irritants and friction, and simplify the lives of users. 
In reality, their incentive is quite the opposite: create, spark, 
generate an irritant or discomfort to be then processed ... 
A pop-up that is triggered when reading the article to ask 
them to log into the site (as explained in our boxed item) is 
a good example. 


15 lan Bogost, The Cigarette of This Century, The Atlantic, https://www.theatlantic.com/technology/archive/20 1 2/06/the-cigarette-of-this-century/258092/ (viewed on 18/12/2018) 
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Making us addicts? 


Addiction to screens is a debate that tops the list of concerns 
of the media and the authorities. Yet, like discussions that 
focus on video game addiction, opinions do not naturally 
converge towards a consensus. 


For the anthropologist Natascha Schüll, a professor at the 
University of New York, there are clearly identifiable parallels 
between the mechanisms set up by the gaming industry 
(especially slot machines in casinos, studied in her latest 
book!°) and methods developed on the internet”. The resear- 
cher describes, in the case of slot machines, states close to a 
form of trance, which she calls the “machine zone”; moments 
in which everyday concerns, social demands and even body 
awareness disappear. These states are partially applicable 
to the relationship with digital tools: “In the online economy, 
revenues are a function of the attention of consumers, 
as measured by the click-through rate and time spent. [...] 
Whether the goal is to win emojis on Snapchat (Snapstreak), 
scrolling images on Facebook, or to play Candy Crush (for 
which we explained the retention mechanisms in our IP3 
Report), you are caught up by fun loops or cycles based on 
uncertainty, anticipation and reactions whose rewards are 
just sufficient to incite you to carry on”. Jaron Lanier, a former 
computer scientist at Microsoft, agrees with this analysis 
when he states that “we have been gradually hypnotized by 
technicians whom we do not see, for goals that we do not 
know, like laboratory animals'®”. 


Nir Eyal addresses the issue of risk involved in voluntarily 
creating addiction, a risk which he says affects only a small 
number of users, as individuals have an increasing ability to 
self-regulate with the (benevolent) assistance of companies: 
“companies [..] have a moral obligation and, maybe one day, 
a legal obligation to inform and protect users that produce 
an unhealthy attachment to their products”. 


Here we find a kind of cyber health-awareness very cha- 
racteristic of Californian culture. As pointed out by Nathan 
Jurgenson, it is difficult to define what would correspond to 
healthy practices, tree of signs of addiction: “The assumption 
is that some users are sick and others healthy. Who decides 
what is healthy? What do people do with their phones? At 
best, they talk to each other. Sometimes, admittedly, just 
to maximize figures. But can you be addicted to talking to 
people? To being sociable? | don’t think so. For me, talking 
about “sick” and “healthy” users leads to a rather scary and 
conservative normalization, even though | do not believe that 
is the intention’. 


This question of a real or supposed addiction to digital ser- 
vices also raises the question of its regulation, including by 
States. As we will see later, the digital giants already offer 
control tools, which carry within them more questions than 
real solutions. 


16 Schill, Natascha, Addiction by design: machine gambling in Las Vegas, 2012 


17 Mattha Busby, “Social media copies gambling methods ‘to create psychological cravings", 
The Guardian, 8 mai 2018, 

https://www.theguardian.com/technology/2018/may/08/ 
social-media-copies-gambling-methods-to-create-psychological-cravings 

(viewed on 06/12/2018) 


18 Jarod Lanier, Ten Arguments for Deleting Your Social Media Accounts Right Now, 2018 


19 Annabelle Laurent, “ Sur les réseaux sociaux, le contenu n'existe que pour maximiser les likes ', 
Usbek & Rica, 30 September 2018, 
https://usbeketrica.com/article/reseaux-sociaux-contenu-existe-maximiser-chiffres-nathan-jurgenson 
(viewed on 06/12/2018) 
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TOOLS THAT TRANSFORM INTO BENEVOLENT GUIDES 


Nudge as a positive vision 
of manipulation? 


The nudge, which can be translated as a gentle incentive, 
is, according to Wikipedia, “a technique to encourage indivi- 
duals or a target population to change their behaviour or to 
make certain choices without being under constraint or an 
obligation and does not involve any sanctions”. 


A topic popularised from 2008 by Richard Thaler and Cass 
Sunstein?’ as an extension to the critique of the rational 
economic agent, this technique consists 
in influencing behaviours towards what 
are considered to be positive objec- 
tives. In this perspective, as Norman 
points out?’, designers should bear in 
mind that users of their objects are 
human, confronted daily with a myriad 
of choices and signals to be processed. 
The aim is therefore to act on the archi- 
tecture of individuals’ choices to encou- 
rage them - we often talk about gentle 
incentives - to take certain actions 
rather than others. 


u“ 


Architects of choices accept the fact of influencing the 
choices made by human beings, to induce beneficial beha- 
viours (for the individual, the community or the planet), in a 
flaunted paternalistic view. To the contrary, improving the 
business model of a company or service is not a nudge as 
designers would have it, but would amount to incitement at 
best and manipulation at worst??. 


Relieving us of informational overload, 
for real choices? 


Humans are cognitively used to relying on the expertise of 
others. You do not necessarily know how to make a musi- 
cal instrument or lamp - someone else has designed these 
objects so that you may use them”. Interface designers 
and developers design the architecture of services to allow 
browsing and, where applicable, make choices for ourselves 
in a complex digital ecosystem. 


Cass Sunstein compares 
the Nudge to GPS: it lets 
you go where you want to 
go but tells you the right or 
best way to doit. j 


Hubert Guillaud, InternetActu 


Thaler and Sunstein, in particular, highlight the orientation 
of individual choices towards solutions that offer the least 
resistance: “due to laziness, fear or distraction, people will 
tend to choose the option that will require the least effort, 
or the route that will offer the least resistance” (Sunstein). 
Individuals will thus tend to always choose the “default” 
option, regardless of whether it is good or bad. Where the 
supporter of gentle incentive will always strive to integrate 
best practices in “default” versions, which falls in line with 
the obligation to “privacy by default” of Article 25 of GDPR, 
some may however be tempted to use this bias for less 
positive purposes. 


In terms of data protection and res- 
pect for privacy, cognitive overload 
issues are equally important for our 
agency as interfaces that guide us in 
our choices. Information overload is one 
of the cognitive biases that lead us to 
make choices without controlling all the 
cards we have been dealt. In his book, 
Choosing not to choose, Cass Sunstein 
develops an ambitious political theory 
of choice and its architectures. 


His theory is that having a choice is conceptually seen as 
a positive challenge in all circumstances because it stren- 
gthens the individual-king, when in fact true freedom is 
sometimes linked to the power of not having to choose. 
Choosing perhaps a burden, as time and attention are pre- 
cious, rare resources. Choosing not to choose can be a 
way to increase well-being and freedom, provided you have 
confidence in the system set up and are consistent with its 
objectives; these conditions are not self-evident and need 
to be shared, to be transparent and clear. 


20 Cass Sunstein et Richard Thaler, Nudge: Improving decisions about health, 
wealth and happiness , 2008 


21 Ref (op cit) 


22 Hubert Guillaud, “ Où en est le Nudge (1/3) ? Tout est-il “ nudgable " ? ”, InternetActu, 
http://www.internetactu.net/20 17/06/27/ou-en-est-le-nudge- 13-tout-est-il-nudgable/ 
(viewed on 06/12/2018) 


23 Steven Sloman et Philip Fernbach, The Knowledge Illusion: Why We Never Think Alone, April 2017 
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Focus on... 


Example of a nudge 


Setting up nudge techniques to slow down moto- 
rists, for example by painting pedestrian crossings 
so that they seem raised will have no impact on 
the individual freedoms of motorists who will simply 
slow down when approaching a pedestrian cros- 
sing. But in parallel, this nudge has an essential 
positive impact on the safety of pedestrians who 
need to cross the road. 


Beyond this vision and pragmatic approach, we are not 
always able to choose with the facts in hand, that is to say 
take into account all the information linked directly or indi- 
rectly to this decision. Even a full reading of all the privacy 
policies and terms of services that we accept (which would 
take at least 25 working days per year”) would only give a 
fragmentary view of all the ins and outs related to a service. 
It is therefore important to shape the information and archi- 
tectures of choice so that individuals are properly guided. It 
is the mission of interface designers to stage these choices 
virtuously and not misleadingly. 


If the intentions of the nudge strive to be positive and pro- 
mote general interest, the very term of soft paternalism 
adopted by its designers raises the issue of the freedom 
of individuals to exercise their own choices. The issue of 
individual autonomy and the ability to accept or not certain 
injunctions must be raised whenever the goal is to subtly 
encourage certain actions that the individual would not have 
wanted to take. 


Moreover, as pointed out by Célia Hodent (member of the 
CNIL Foresight Committee), the problem with the nudge or 
dark pattern stance is that they paint everything white or 
black, while we face many shades of grey... 


24 Aleecia Mac Donald et Lorie Cranor, “ The cost of reading privacy policies *, Journal of law and policy for the information society, vol. 4, n°3, 2008 
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interfaces 


“The purpose of education is not to get men to admire 
ready-made legislation but to enable them 
to assess and correct it”. 


Nicolas de Condorcet “ Sur l'instruction publique ” (1791-1792) 
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The efficiency of design techniques, in terms of capturing 
the attention of Internet users and directing their behaviour, 
necessarily leads us to look at structures that implement 


these strategies in the most striking way, the most prominent 
being the great data industrialists. 


The choice of platforms in terms of design of their interface 
and services plays an important role in defining the field of 
possibilities (for the available features or not), actions (which 
can be encouraged or, on the contrary, made more difficult) 
and ultimately users’ preferences (since we tend to prefer 
what we are used to). 


Unsplash cc-by Vladislav Klapin 


Platforms’ leverage capacity is particularly important when 
their audience is extremely broad and loyal, because they 
capture a large proportion of users on their interfaces. This 
obviously does not mean that this power is based on a 
necessarily manipulative use of techniques, but it is clear 
that the economic model of these platforms is usually largely 
underpinned by revenues from advertising on which partly 
rely - the services actually offered also explains it - on effec- 
tive internet user retention techniques already described in 
this document. 


Every month, nearly a third of humanity (2.27 billion people) 
connects to Facebook (for a monthly average of 18 hours 
and 24 minutes?) and 6 billion hours of videos are viewed 
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on YouTube. Amazon makes more than 5 billion deliveries 
per year. For all media content put together, in 2018, the 
French spent on average 4 hours and 48 minutes per day 
on the internet, including almost one and a half hours on 
social networks? 


Alongside the direct effect exerted on their users, the big 
platforms are also able to use their central position in the 
digital ecosystem to establish themselves as essential refe- 
rences for all of their sector, through standards used by all. 


l 
These [design] tools 


are therefore capable 
of transforming into particularly 
effective soft power levers as 
they fashion the digital 
world to the model 
of big platforms. s 


The case of the kml format (used for geolocation) created 
by Keyhole Inc., and acquired by Google in 2004, is an 
example in this respect, as it has become the norm. KML 
is now the most frequently used format for all geolocation 
tools. Google's Material Design”, meanwhile, has adopted 
a B2B influence strategy. On a dedicated platform, and on 
the basis of work carried out largely in-house, the company 
provides advice and tips to “support innovators in [their] 
area’. This project, launched in 2014, proposes to merge in 
a graphic and ergonomic charter “the classic principles of 
good design with technological and scientific innovations”. 
By making available various tools such as icon systems, or 
colour palettes, the aim is to enable the development, accor- 
ding to Google, of a “single base system that harmonizes the 
user experiences across platforms and devices”. 


Made accessible to others in the form of directly usable kits, 
these tools are thus capable of transforming into particularly 
effective soft power levers, as they fashion the digital world 
to the model of big platforms. 


25 Connie Hwong, Verto Index: Social Media, 
https://wwwvertoanalytics.com/verto-index-social-media-4/ 


(viewed on 18/12/2018) (viewed on 18/12/2018). 


26 Thomas Coëffé, Internet Figures — 2018, Blog du Modérateur, 
https://www.blogdumoderateur.com/chiffres-internet/ 


27 https://material.io/design/ 
(viewed on 18/12/2018) 
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Warnings from the 
Tech Repentants 


Repented Silicon Valley figures are increasingly acting as 
whistleblowers against attention-capturing strategies imple- 
mented by companies. For them, the technologies they 
helped design cause both individual and social problems 
which the platforms are unlikely to want to solve since they 
benefit from them economically. 


Speaking about the “like” feature, Chamath Palihapitiya, a 
former Facebook executive, believes that “short-term fee- 
dback loops stimulated by the dopamine we have created 
are destroying the way our society works” For him, the best 
solution is abstinence (which is probably unrealistic in today’s 
society) because he recommends not to use these services: 
“if you feed the beast, it will destroy you?®”. 


Others seek to make technology more ethical, like the 
Center for Human Technology founded by Tristan Harris, a 
former Google employee. With the objective of “realigning 
technology with the best interests of humanity’, this orga- 
nisation seeks to raise public awareness about so-called 
dangers of tech; it promotes protective design against our 
inherent vulnerabilities (cognitive bias, etc.) and encourages 
policy initiatives that move in this direction. The aim is to 
avoid the “erosion of the pillars of our society’, i.e. “mental 
health, democracy, social relations and children’, which the 
“race for monetization of our attention” is compromising. 


The stances of tech repentants should however be taken 
with caution as they only feed the belief of the omnipotence 
of companies which would be futile to attempt to regulate. 
When Jaron Lanier argues” for the disconnection of social 
networks that “bring out the worst of human nature, and 
make us aggressive, egocentric and fragile” rather than 
actually encourage the exodus, it is possible that this stance, 
on the contrary, promotes the impression of the impotence 
of users and the impossibility of controlling these platforms 
to ultimately let them regulate themselves... The response 
of the GDPR is rather to give users the means to control 
the use of their data, by wagering on responsible innovation 
rather than self-regulation. 


Try-fail-fix as 
a design method? 


Growing concerns about the influencing capacity of plat- 
forms tend to surreptitiously result in stances on the omni- 
potence of the big tech companies whose strategies are 
perfectly orchestrated. 


Yet this Promethean, if not “conspiracy theory”, narrative 
comes up against the realities of development models of 
these services, which, based on often experimental tech- 
niques, frequently reveal their fallibility. Sometimes claiming 
that they do not really know what they are doing or why, the 
big platforms also claim that they are incapable of explaining 
why things work or do not work as expected ... Which is not 
necessarily reassuring. 


Some decisions taken by platforms, especially those related 
to how to design and present interfaces, are largely ins- 
pired and influenced by the reactions of users and their 
ways of taking control of the tools that are offered. Thus, 
the Facebook wall was initially a very limited feature. You 
had to go on a user's page to view his/her wall and it was 
not until 2011 that the “news feed” feature was introduced. 
It is because users interacted, hijacked and played with the 
wall that it was changed. Similarly, the famous Twitter hash- 
tag was invented by users and not by the company which 
happily and smartly draws the benefits of this creation. This 
development by trial and error is facilitated by the ability to 
implement life-size experiments on a large pool of guinea pig 
users. Access to captive panels has allowed Google to test 
40 different shades of blue for its hyperlinks, or the OkCupid 
dating platform to measure the real impact of its “match 
percentage” by pretending to members that they were very 
compatible when they had little in common ... (see box). 


These experiment strategies are symbolic of the iterative 
approach that characterises the big platforms. Gretchen 
Sloan, a Facebook spokesperson, explained in an article 
that it is “a very common approach [..]: launch a product / 
feature, see how people use it, and then improve it over time. 
This helps us (like other companies around us) to design and 
quickly implement new features that people want”*°. 


28 Guillaume Ledit, For a former Facebook manager this “shit” 
“destroys the social fabric of our societies” 

Usbek & Rica, December 2017 

(viewed on 18/12/2018) 


Accounts Right Now, 2018. 


29 Jaron Lanier, Ten Arguments for Deleting Your Social Media 


30 Rachel Kraus, “Facebook's “Time Management’ tool shows 
it hasn't stopped treating users like psychological guinea pig’, 
Mashable, 1st August 2018. 

https://mashable.com/article/ 
facebook-instagram-time-management-psychological-tests/ 
(viewed on 29/11/2018) 


It does not seem advisable for society to turn itself into a 
giant, open-cast behavioural psychological experiment wor- 
king for large web companies, and this is why serene and 
responsible regulation is important: because we cannot only 
settle for corrections through new test iterations and (new) 
errors. 
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Zoom on... 


OkCupid or the love laboratory 


The OkCupid dating site received a lot of attention 
in 2014 for having described on its blog various 
experiments conducted on its users (the post in 
question has been removed, but studies were also 
reported in Dataclysm, written by Christian Rudder, 
one of the founders of the platform). Two of them 
attracted particular attention: 


¢ Manipulation of the match percentage. To 
know the real impact of its match percentage (com- 
patibility between two members), the OkCupid tra- 
demark, the platform artificially modified its values 
for different users. 90% compatible people showed 
a 30% match and vice versa. The result? When 
users thought they matched, they liked their dis- 
cussions more and conversations lasted longer... It 
gives food for thought about the power of sugges- 
tion of platforms. 


e “ Love is blind ”. OkCupid allowed users to 
give scores to the profiles of other members. By 
masking the blurb of certain profiles, which then 
only featured a photo, the platform realised that the 
description only accounted for 10% of ratings. Love 
is not apparently as blind as all that, at least not on 
OkCupid... 


The author justifies this practice by claiming that 
“OkCupid does not really know what it is doing, 
nor do the other websites. It is not as if people 
were trying to build these things for a long time, 
or as if you could refer to a specific plan. Most 
ideas are bad. Even good ideas could be better. 
Experimenting is the only way to settle it all.’ 


| 29 


31 Christian Rudder, Dataclysm, 2014. 
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Limits of self-regulation 


In fact, today, whether for fake news or attention manage- 
ment, platforms are presented - or present themselves - as 
the perfect antidote to remedy the ills they generate. 


To respond to criticism and charges of “theft of attention” 
or “time theft’, in the words of Tristan Harris, the major market 
players in 2018 launched their own time spent management 
tool on their different applications. 


In a quantifiedself approach, the Google Android P tool offers 
users a dashboard that allows them to see the time spent 
on their phone, the number of notifications received, the 
breakdown of time spent per application and their hourly 
use. The amount of captured data is however absent from 
this dashboard. The application then suggests the user to set 
time limits for each use, with alarms, like a sort of parental 
control that we would apply to ourselves. Apple, Facebook 
and Instagram launched similar tools between June and 
August 2018. The benefits of these tools will be assessed 
over time, especially as a Thai study has shown that people 
most using self-regulatory tools were also the most likely to 
have behaviours close to addiction*?. 


A credible economic hypothesis is that these tools are 
consistent with their strategy as custodians of attention as 
these players are essential for advertisers, acting like tolls 
to access our attentional devices. We should therefore not 
underestimate the effectiveness of a commercial stance 
consisting in telling advertisers: “I know when the individual 
must not be disturbed as they use my attention management 
tools; | am therefore the only one to avoid your brand being 
considered as an irritation”. 


By leaving platforms to regulate their own shortcomings - in 
the tradition of communication on their own responsibility, 
as a form of apology - these solutions also act as a form of 
transfer of responsibility from structures to individuals. As 
said Antoinette Rouvroy in the foreword to our IP2 Report, 
these logics tend to make individuals entrepreneurs of their 
well-being, solely responsible for their bad habits, thus absol- 
ving economic players or society of their responsibilities. If 
you succumb to the cognitive or emotional “hooks” of desi- 
gners to the point of suffering consequences close to patho- 
logical addictive states, it is ultimately something of your own 
fault. The proof is that we give you the tools to self-regulate! 


Thus, the digital world is both the poison and the remedy 
according to the classic analysis by Bernard Stiegler, and 
this “Pharmakon” logic ® has never been so explicitly incor- 
porated into economic models. But in this case, it is in a logic 
essentially assigning to the individual the responsibility of 
finding doses that separate one from each other, with the 
supposed kind help of behavioural crutches produced by 
the very people that build attentional traps in the first place. 


32 Vikanda Pornsakulvanich, “Excessive use of Facebook: The influence of self-monitoring and 
Facebook usage on social support’, Kasetsart Journal of Social Sciences, 
https://www.sciencedirect.com/science/article/pii/S24523 15116300819 

(viewed on 07/12/2018). 


33 In ancient Greece, the pharmakon refers to both the remedy, poison and the scapegoat. 
http://arsindustrialis.org/pharmakon 
(viewed on 07/12/2018). 
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WHEN PERSONAL DATA COLLECTION 


BECOMES BIASED 


Bad design, dark patterns 
and personal data 


When the different previously presented techniques are 
implemented with the aim of accumulating more data on 
individuals, customers or citizens than necessary, they do 
not only pose questions of ethics and the responsibility of 
digital services with respect to capturing attention. They also 
confront the basic principles of the GDPR that gives indi- 
viduals greater rights on the use that is made of their data. 


Several authors have addressed this issue, as well as 
Norwegian consumer protection associations (see below), 
and of course the CNIL. As we have seen, individuals are 
confronted with biases that can have various instruments - 
something that certain players have understood - and these 
can have significant impacts on data protection. Woodrow 
Harztog ranks these practices into three categories, each of 
which can contravene the regulations, but at varying levels 
for users! 


e Abusive design : uses the limitations and cognitive biases 
of individuals to get them to perform actions over which they 
have no control. Whether through dark patterns (see below), 
attention retention techniques, or even the use of difficult to 
understand jargon, vague terms or double negatives, all are 
“magic” techniques that will be used to influence or mani- 
pulate users. 


e Deceptive Design : refers to practices aimed at represen- 
ting elements in such a way as to mislead the individual. For 
example, the use of privacy protection indicators, such as 
specific logos, icons or badges, without the service actually 
being virtuous or secure. It also refers to deception by omis- 
sion, when an application collects certain data without the 
user being aware of it. 


e Dangerous design : this corresponds to methods that will 
make us vulnerable, either directly or indirectly. 


Another approach is the one put forward by Harry Brignull 
in 2010 with the concept of dark patterns: in their search to 


capture individuals’ attention and collect even more of their 
personal data, platforms and interface designers of digital 
services have created deceptive models for individuals which 
act on psychological phenomena specific to each one of us. 
From the perspective of the protection of privacy, there are 
several types of users traps and abusive design, described in 
several works including those by Harry Brignull**, Norwegian 
consumer associations (in their report entitled Deceived By 
Design * ) or Lothar Fritsch (University of Darmstadt). 


The onus lies with us to propose a non-exhaustive typology 
of these practices, which have a direct impact on data pro- 
tection, which you will find on the next page. 


These practices can affect the ability of individuals to 
effectively protect their personal data and make conscious 
choices. In addition to privacy policies that need to be com- 
plete and in compliance from a legal perspective, it is impor- 
tant not to overlook the implementations and staging of the 
different moments in which interface designers seek to 
influence individuals. We cannot be satisfied with the words, 
“non-contractual photograph”, like wording featured on food 
packaging. Packaging this time has a direct impact on the 
rights of individuals and should be taken into account in 
assessing compliance of the entire service: we will discuss 
such proposals in the next section... 


34 http://darkpatterns.org/ 
(viewed on 07/12/2018) 


35 Forbruker radet, Deceived by Design — How Tech Companies use dak patterns to discourage us 
from exercising our rights to privacy, https://fil forbrukerradetno/wp-content/uploads/2018/ 
06/2018-06-27-deceived-by-design-final.pdf 

(viewed on 07/12/2018) 
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A NON-EXHAUSTIVE TYPOLOGY 
OF POTENTIALLY DECEPTIVE DESIGN PRACTICES 


We classify these practices into four categories (and columns) from a data protection perspective for which different 
design tactics can be implemented: enjoy / seduce / lure / complicate / ban. Some of these practices may comply 


with the GDPR but, depending on the time, manner and data in question, they can raise ethical issues and even be 


non-compliant. 


PUSHING THE INDIVIDUAL TO ACCEPT 
SHARING MORE THAN WHAT IS STRICTLY 
NECESSARY 


- SAFETY BLACKMAIL [ENJOY] 

At the login, requesting additional information to what is 
strictly necessary for the service in situations where users 
are under pressure, when they have just entered or renewed 
their password, updated their profile information or placed 
an order. The user is engaged in a process that they rapidly 
want to conclude and will tend to accept anything without 
taking the time to analyse the request, especially if it is tied 
to a (real) need for security. For example, getting the user 
to believe that giving their phone number will be used for 
delivery, or two-factor authentication, when it is only used 
for telephone prospecting purposes. 


- JUST BETWEEN YOU AND US [SEDUCE] 
Requesting additional and not strictly necessary data for 
the execution of the service with the promise that such data 
remains “invisible” and under the user's control or will allow 
a better service, for example when a social network asks 
you to complete information on your past life, the school you 
attended or sports club in which you were enrolled. 


- FALSE CONTINUITY [LURE] 

Asking the user to give their address in order to read the 
article (title) without giving enough clear warning that this 
is actually a subscription to a newsletter (or in such small 
writing it cannot be read). 


- IMPROVING THE EXPERIENCE [SEDUCE] 
Using the customisation and improved user experience argu- 
ment to encourage the user to share more data. 


- DEFAULT SHARING [ENJOY] 
Pre-checking information sharing options, which will not 
always be unchecked when signing in. 


INFLUENCE 
CONSENT 


- TRICK QUESTION [LURE] 

Writing a question in such a way that quick or inattentive rea- 
ding can lead you to believe that the answer option produces 
the opposite of what you think you are doing. For example, 
use of a double negative can lead to accept a refusal … For 
example, the button accept is subtitled “Yes, send me the 
food programme” when the refuse button said “No thank 
you, | do not like delicious food” 


- LAST MINUTE CONSENT [ENJOY] 

Seeking consent for the collection of data at a specific 
moment where we know that the individual is in a weak 
position because in a hurry or impatient to finish. For exa- 
mple, integrating a prospecting opt-in with partners in the 
final stages of the confirmation of an order... 


- ATTENTION DIVERSION [ENJOY] 

Drawing attention to a point of the site or screen to distract 
you from other points that could be useful. For example, 
working on the colour of a “continue” button while leaving 
he “find out more” or “configure” button smaller or grey. 


- COMPARISON OBFUSCATION [COMPLICATE] 

aking comparisons difficult: between one service and ano- 
her, or when there are changes in the settings or rules. For 
example, changing the formulations on content privacy / 
advertising settings on a social media so that the user does 
not easily adopt a permanent routine to reduce the visibility 
perimeter of these publications. 


- WRONG SIGNAL [LURE] 

Using a “universally” understood graphic code to mean the 
opposite, thus creating a confusion for the user about the 
choice they are making. For example, adding a padlock to a 
not especially secure interface. 


CREATING FRICTION ON 
DATA PROTECTION ACTIONS 


- BLAMING THE INDIVIDUAL [ENJOY] 

Make the user feel guilty about their choices, by the words 
used. This is very often used for example by media whose 
business model is based essentially on advertising, when 
a user refuses advertising tracking or uses an ad blocker. 


- IMPENETRABLE WALL [BAN] 

Blocking access to a service by a cookie wall or account 
creation while it is not necessary to use the service as such 
(also called take it or leave it). No alternative without tracking 
is available. 


- MAKING IT FASTIDUOUS TO ADJUST 
CONFIDENTIAL SETTINGS [COMPLICATE] 

Facilitating consent by a simple action and making the pro- 
cess of data protection longer and complicated. For example, 
allowing simple continuity to accept all opt-ins (a “continue” 
button) while the advanced options and settings involve a 
winding alternative path, consisting of “find out mores” and 
scroll bars. 


- REPETITIVE INCENTIVE [COMPLICATE] 
Insert incentives, during the user experience, on data sharing 
requests to repeatedly interfere in the pathway. 


- OBFUSCATING SETTINGS [COMPLICATE] 

Creating a deliberately long and tedious process to achieve 
the finest settings or make them so fine and complicated 
that they will encourage the user to give up before reaching 
their initial target. 


SHAPING CHOICES IN THE DIGITAL WORLD 
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DIVERTING 
THE INDIVIDUAL 


- BAIT AND CHANGE [LURE] 

A setting or a choice made by the individual produces a 
different result than desired. For example, giving accep- 
tance value to a button with a cross, which in users’ minds 
is synonymous with “close and move on” This method has 
for example been used by Microsoft to “encourage” users of 
the previous version of its Windows OS to switch to Windows 
10. Due to public reactions, Microsoft acknowledged it had 
made a mistake and backtracked®®. 


- CHAMELEON STRATEGY [LURE] 

A third party service takes on the style and vision of the 
website where you are browsing to make it a natural conti- 
nuation of a process. For example, a service is added to an 
onboarding or train ticket ordering process to tie in a car 
rental with a trading partner. These strategies are also found 
in software installation... 


- CAMOUFLAGED ADVERTISING [LURE] : 

Advertising is disguised as another type of content or ele- 
ment of the interface, in the hope that the user clicks without 
knowing that it is advertising. 
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Consent, a free will that 
is far from illusory 


The advent of the GDPR and the over-valuation of consent in 
the way it has been presented have created the emergence 
of strong criticism of the notion of consent. 


The psychologist Barry Schwartz had theorised the para- 
dox of choice (The Paradox of Choice, 2004): “even 
though autonomy and freedom of choice are fundamental 
human values, too many choices and too much control can 
overwhelm us and lead us astray.’ 


Helen Nissenbaum, professor of infor- 
mation sciences at Cornell Tech, goes 
further by pointing what she calls the 
farce of consent that lulls users into 
a false sense of control: “Even if you 
wanted to create totally transparent 
consent, you could not”. In her view, 
even the best-intentioned companies 
do not know what happens to the data 
they collect. While it is true that the 
consent is not always informed, the 
problem with this criticism lies in the 
fact that it can disempower companies 
with respect to the data they process. 
Yet the GDPR is clear: they are duty- 
bound to know how data is used and 
required to set up protection means for individuals, mapping 
and securing data as well on the management of the data 
life cycle. Giving consent does not amount to signing a blank 
cheque to a company or organisation, but rather imposing 
on it certain rules of respect for the rights of the individual. 


u“ 


On the link between interface designs and the choice of 
individuals, Woodrow Hartzog” stresses that if we do not pay 
sufficient attention to the design of technologies and inter- 
faces, we could bring all the consequences of design choices 
to bear on individuals alone. The fetishisation of control is, in 
his opinion, one of the major weaknesses. Some companies 
will ostensibly give their users all possible settings options 
to then claim that the design of the interface is privacy and 
user friendly. The accumulation of choices can overwhelm us 
and distract us (from the essentials). Choice “then becomes 
an illusion of empowerment and is transformed instead into 
a burden”. The researcher thus criticises excessive focus 
placed on the manufacturing of consent, taken up by plat- 
forms - including GAFAM - for whom all privacy protection 
problems could be solved by giving more control to users, 


Giving consent does not 
amount to signing a blank 
cheque to a company or 
organisation but rather 
imposing on it certain rules 
of respect for the rights of 
the individual. ” 


when real control does not grow in proportion to the mul- 
tiplication of possible choices and where consent is not a 
sort of joker among data protection principles allowing them 
to do anything. 


However, the aim is not to throw the consent out with the 
bath water, firstly because it is not the only legal basis for 
data processing, but also and mainly because consent 
always occurs, despite what Nissenbaum and Hartzog say, 
in aconstrained space. The real risk is not that of consenting 
in itself but to believe in and to uphold a principle of abso- 
lute informational self-determination in which each individual 
would be in control and, in particular, responsible for all their 
actions. Consent, if it must be freely 
given, specific, informed and unam- 
biguous (according to Article 4 (11) 
of the GDPR), nevertheless remains 
constrained in its scope and its related 
legal obligations. Some speak of rea- 
sonable approximations, or reasonable 
fiction, but it would be preferable to talk 
about consistent scope of responsibility 
and control by individuals. The consent 
given by an individual does not exempt 
a service provider from complying with 
all applicable rules, including security, 
loyalty, transparency, limitation of out- 
comes as well as all user rights. 


Free consent is for the user a weapon to protect their rights, 
insofar as it remains a legally binding principle: it is possible 
to argue, debate and decide on the validity of how it is col- 
lected... As such, the fact of using and abusing a strategy 
to divert attention or dark patterns can lead to invalidating 
consent. 


37 See: https://news.softpedia.com/news/microsoft-admits-it-went-too-far-with-aggressive-windows- 10-updates-511245.shtml 


(viewed on 6/12/2018) 
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Design could provide 
answers to these issues 


Rather than having a defeatist attitude and challenging the 
very notion of consent and invoking the ability for individuals 
to act with full awareness of the issues raised by the col- 
lection of the data, it would be better to look at how players 
could use solutions offered by design, not to hide, obfuscate 
or subtract, but rather to highlight and positively support 
users in understanding the mechanics of digital services. 


In 2013, law professor Ryan Calo, in his article, Against 
Notice Skepticism**, proposed that design could be used 
and implemented to inform in a way that allows more 
enlightened consent of individuals. Solutions and responsibi- 
lities must not, in his opinion, always involve individuals: “You 
can put up posters all over the city to remind pedestrians 
that electric vehicles are silent, or you can ask carmakers 
to introduce a motor sound in their vehicles”; similarly, “you 
can write complete and very lengthy privacy policies that 
few people read, or you can design your site (or applica- 
tion) so that users can be on their guard when their data is 
collected or else be able to demonstrate how their data is 
effectively used”. If these two such principles should not be 
so categorically opposed, it is also in the user experience 
that information should be disseminated. This would allow 
users to act consciously and understand, while protecting 
themselves from informational overload that could lead to 
the temptation of the exhaustive confidentiality policy. If the 
latter must always be present, as a reference point, it must 
be accompanied by the design. 


The lawyer and designer Margaret Hagan, director of the 
Legal Design Lab at Stanford University has highlighted the 
necessary convergence between legal and design issues. 
In an interview with Facebook TTC labs*°, she said that 
although the law is born from human experience, it is not 
always thought out in terms of experience, with a focus on 
how people think, what they feel and understand. According 
to her, people want to remain strategists, understand the 
options they are presented and protect their rights, but too 
often, she believes “the legal system produces the opposite 
effect, people do not trust and may feel that they have no 
power”. This is where the designer's work must intervene 
to produce, with the regulator, the methodologies that will 
ensure stronger trust of users. The CNIL has already started 
this work and we will describe it in the next section. 


38 M.R. Calo, Against Notice Skepticism in Privacy (and Elsewhere), 39 https://www.ttclabs.net/insight/why-law-needs-design 
87 Notre Dame L. Rev. 1027 (2013), (viewed on 20/12/2018) 
http://scholarship.law.nd.edu/ndlr/vol87/iss3/3 
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The media that lives in the future 


> 2030 
THE NEUROTARGETERS 
ATTACKS 


"Creating desire” is no longer an 
expression but a reality. 


> 2070 
THE ADVENT OF 
SUPERPOWERS 


Will science make us invincible? 


S EEATURE = 


ON THE 
NEUROTARGETERS TRAIL 


In just 10 years the digital landscape has changed drastically. Monitors and keyboards have been 
replaced by surfaces and voices. Following this transformation characterised by a kind of absence 


of devices, the advertising world has learned to overcome all barriers to make us desire on demand. 


Here is an overview of a new practice directly connected to your brain. 


ADA ROY: 
“MY ROLE IS 

TO OPTIMISE 
YOUR BRAINS 
NEURONAL 
CONNECTIONS’ 


Ina French novel that I read as a student, a mysterious «7th 
language function» could give people the almost absolute 
power of conviction. The author had semiologist Umberto 
Eco say that: «the person with the knowledge and control of 
such a function would virtually be the master of the world. 
Their power would not have no limits. They could be elected to 
all positions, raise crowds, cause revolutions, seduce women, 
sell all sorts of imaginable products, build empires, obtain 
anything they wanted in any circumstances. ». The magic of 
language devised by Laurent Binet in 2015 seems to be about 
to become reality with neurotargeting. Wishing to explore 
this practice on the verge of esotericism, I met Ada Roy, one 
of the pioneers of this discreet industry and co-founder of 
SKIN. Who knows what effect this so-called confession will 
have on you... 

By LAURA HACHECROIX 


Although it was imaginable that targeted advertising 
was going to disappear with screens, SKIN is the per- 
fect example of the opposite. How have you managed 
to adapt to this radical transformation of the digital 
ecosystem? 


Ada Roy: What our business has in common with the 
marketing and advertising professionals of the 2010s are 
digital tracks. The similarity ends there. While in 2010, 
the idea was only to target customers, nowadays we must 
design a product, pricing and advertising strategy for 
each customer! 


To accomplish this, the boxes of socio-cultural and 
professional profiles such as «man, aged 35-45, city 
dweller with 2 children working in insurance, likes 
reggae, surfing and cooking a la plancha», generated 
from people’s online activities, are no longer enough. The 
data we now want is the one that will reflect in the digi- 
tal world the subtle chemistry of consumers’ neurons. 
This is where we, neurotargeters, come in. We track their 
emotions, personality, brain activity. We imagine their 
brain state, and predict their dopamine and adrenaline 
levels. We monitor their thought patterns to instil the 
right purchasing trigger, at the right time. In short, we 
map their neurotransmitters to bring them to our client. 


Is it not rather surprising for a neuroscience Ph.D. 
to move to the world of neuromarketing? 


It came very naturally, following on from my research 
on the relation between neurotransmitters and our daily 
habits. I often compare my work with that of an electri- 
cian: my role is to optimise the electrical connections of 
your brain. In the twentieth century, marketers would 
buy available brain time. Today, I tend to sell available 
brain portions. When I say sell, let me reassure you, I 
mean rent and SKIN is a very discreet temporary occu- 
pier: it hardly disturbs anything, just a few connections, 
here and there. But each rental leaves a mark: the brain 
is a malleable organ, nothing that takes place there is 
definitive, but nothing is trivial either. 


TCHI IA 
watches over your privacy 
PRIVATE RELIABLE SMART 


your 
tificial intelligence li 


your preferences 


à adapts to all char 


You're not alone on the highly competitive braintech 
market. How do you explain the huge success of SKIN 
against your competition? 

When I look at the progress made since the first tests 
when we were only inserting the client’s brand colours in 
the graphic design of third party services, I’m surprised 
at all we’ve achieved. Our first projects were patchy, 
mainly aimed at impressing investors on the basis of our 
respective doctoral work. For example, we mounted a 
long project to create a sensation of thirst among consu- 
mer segments, at specific times and places: reference to 
heat, using specific colours to filter certain video content, 
discreet integration of references to refreshing drinks 
in written texts, change of tone of voice assistants to 
suggest a dry throat ... we tested everything. Nobody 
really knew what would work or not, but the analysis of 
purchasing acts of targeted people enabled us to show 
real changes... 


One thing led to another and SKIN became like a 
second skin placed on all the «natural» interfaces of 
our clients to reach their potential consumers. It’s the 
advertising chameleon. While many of our competi- 
tors have kept to advertising breaks on voice assistants, 
SKIN makes marketing invisible. Subtly, we will instil 
our markers directly aimed at the target’s brain, at the 
heart of their daily digital habits. While our competitors 
try to spark a desire, we create an irritant, an absence, 
a subtle imbalance, a discomfort, almost imperceptible, 
but perfectly designed and controlled so that our client’s 
product comes to relieve that itch. In this sense, we have 
never been designers of marketing services, but desi- 
gners of neural connections. SKIN is interested in neu- 
rons and not the brain, in neuronal electrochemistry and 
not understanding, in reflexes and not discernment. This 
is the innovative heart of SKIN. 


All this is still quite abstract and feels more like 
a magic trick rather than science. Do you have a 
concrete example to explain how SKIN works? 

The magic of SKIN happens, for example, when the 
voice assistant utters one word rather than another. 
Ironically, other types of neurons that do this work: 
those of machine learning algorithms. They will first 


NOVELTY 


Techno-Luddites, get cooking! Sick of being tracked 


break down the voice of the user as it carries their emo- 
tions. Fear, anger, stress, satisfaction, everything that is 
happening in our brains at the neurotransmitter level 
will end up as signals in our voice that are detectable and 
analysable. While the user innocently asks their voice 
assistant about the day’s weather or asks them to start 
their favourite jazz playlist, we are able to detect fatigue, 
stress or happiness and prepare a nudge adapted to that 
state. In reality, the work of our algorithms does not stop 
at the simple analysis but extends to content generation. 
They will go through dictionaries of synonyms, semiotics 
essays and analyses of lexical fields to choose the most 
appropriate word, but will also look into tone of voice 
and inaudible frequencies to transmit the most appro- 
priate message to the person's emotional context. 
Today SKIN is extending to all senses, thanks to agree- 
ments signed with connected objects platforms to access 
their data and allow us to massively analyse faces, ges- 
tures, respiratory movements, skin temperature... 
SKIN is a hunter of human signs: all tracks are good. 


and hunted by your neural interfaces? The Critical 


Brain Initiative offers you a delicious way to fool them 


with Eat Your Brain. Through carefully prepared dishes, 


disrupt signals read by neural interfaces with your gut. 


This «neuro-recipe» book gives you recipe ideas to get 


your thoughts privacy back! 


Eat Your Brain, 2026, CBI Press, €25,99 
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“Good design means 
that a user's mental map 
of how a technology works 
matches reality” 


Woodrow Hartzog 
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The necessary regulation of design 
and architectures of choice 


= 


ENCRES a Oe |: 


HOW TO MAKE DESIGN AND INTERFACES PART 
OF THE COMPLIANCE ANALYSIS 


The practical implementation by service designers of the 
conditions needed for freely given, specific, informed and 
unambiguous consent raises many questions. Amid informa- 
tion overload and the development of manipulative tactics to 
support economic models driven by commitment, individuals 
are not always able to easily understand the ins and outs, 
the outcomes of data collection, and the use of their data. 
How to reconcile this fact with the cardinal principles of 
the GDPR such as the principle of lawfulness, fairness and 
transparency? 

It is indeed the whole edifice of respect for fundamental 


rights of people which needs to be called into question. 


The protection of personal data is traditionally analysed 
through legal and technical prisms. Similarly, the answers 
given by professionals or by regulators tend to focus on 
these two aspects, which although being fundamental, are 
not enough to rise to the challenges described in the pre- 
vious sections. They do not take sufficient account of the 
interaction space between the individual and the machine, 
the exchange layer between the individual and the proces- 
sing of such data. 


THE NECESSARY REGULATION OF DESIGN AND ARCHITECTURES OF CHOICE 


Getty Image - Oversnap 


Interface design - in the broad sense of the term, from the 
architecture of the service through to the formatting of 
information and consent schemes - is an essential medium 
through which the actual implementation of the regulation 
and the conformity of services in this constrained space 
are played out. 


As we identified in the introduction to this report, the 
concept of privacy by design is often too disjointed from 
the concerns, practices and concepts of design professio- 
nals. Article 25 of the GDPR, which demands the integra- 
tion of appropriate measures of data protection “from the 
design stage”, however logically implies that responsibility 
for conformity is an issue more fairly distributed in design 
processes and that designers should take their rightful place 
and offer their expertise to the protection of users’ rights. It is 
through their action, accountability and better consideration 


SHAPING CHOICES IN THE DIGITAL WORLD 


by the regulators of these rights that privacy by design will 
truly become an operational concept rather than a somewhat 
abstract methodological approach. 


It is time to bring design more directly into a regulatory 
triangle, with legal and technical analyses. Such an approach 
will find its true meaning in the application of the principle of 
transparency, in the expression of consent and in the design 
of the exercise of the rights of relevant people (access, rec- 
tification, erasure, portability, etc.). 


Transparency and Information, 
cornerstones of fair processing 


The European regulation specifies that any processing of 
personal data must be lawful and fair. The fact that personal 
data about individuals is collected, used, consulted or treated 
in any other way, and the extent to which such data is or will 
be processed, should be transparent with respect to the 
natural persons concerned. The principle of transparency 
demands that all information and communication regarding 
the processing of personal data is easily accessible, easy to 
understand and designed in clear and simple terms. 
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In other words, no transparency, no loyalty. As recalled by the 
guidelines on transparency of the European Data Protection 
Board (EDPB)*, “its primary objective is to create trust in 
the processes applicable to citizens by enabling them to 
understand and, if necessary, contest the said processes’ 
Transparency is a legal concept eminently focused on users 
and not on legal aspects. In this, it often seems less practical 
to legal professionals, and may appear, wrongly, as a kind 
of general principle with little scope other than symbolic. In 
reality, here again, as recalled by the European authorities, it 
“is reflected in several articles by specific applicable practical 
requirements” (in particular Articles 12 to 14 of the GDPR). 
Overall, in terms of transparency, the quality, accessibility and 
intelligibility of information are as important as the formal 
content of information provided to the relevant persons. 


The general principle is to present information effectively and 
succinctly, using knowledge that the processing manager has 
of people on which they collect information and the specific 
context of the service they propose. 


Naturally, as consent does not necessarily mean the use of 
a check box, transparency does not necessarily mean an 
exhaustive text. Professionals must use all possible tools, 
interfaces, current and future user pathways: different levels 
of information, FAQs, pop-up windows, conversational agents, 
icons, etc. 


Finally, work on the user path could be put to use by data 
controllers. As highlighted by these guidelines on transpa- 
rency issued by the CNIL and its European counterparts, 
processing managers are recommended to organise user 
tests (with representative panels, for example, or other forms 
of test that are recognised or even normalised, such as legi- 
bility or accessibility) with a view to raising any uncertainties 
on users’ actual understanding. This process of improvement, 
measurement, evaluation and testing can in fact aim to be 
an integral part of the accountability strategy of processing 
managers: the competent authorities could be informed of 
the results of these tests and assess the relevance with 
respect to the principles of simplicity and accessibility of 
information. 


Consent informed by the work 
of designers 


As recalled by the CNIL on its website, consent “ensures the 
relevant persons strong control over their data, by allowing 
them to understand how their data will be processed, choose 
freely whether to accept such processing or not and change 
their mind freely”. 


Consent, according to the European Regulation, should be 
given by a clear positive act whereby the relevant person 
shows their agreement in a free, specific, informed and 
unambiguous way to the processing of their personal data. 
The European Data Protection Board, in its guidelines on 
consent“; states that the adjective ‘free’ implies a choice and 
real control for the relevant persons and that “any pressure 
or inappropriate influence exerted on the person (in diffe- 
rent ways) preventing them from exercising their will shall 
invalidate consent”. 


The European equivalents of the CNIL insist on the res- 
ponsibility of innovation generated by this constraint to find 
new solutions that work according to the scope of the law 
and promote the protection of personal data better, as well 
as the interests of the relevant persons. 


However, it might be considered that the unfair or decep- 
tive design (see above) of digital services can generate 
various problems with consent and is sufficiently objective 
and demonstrable to lead to its invalidity. A person’s control 
of their data becomes illusory, as consent would not be a 
valid basis for its processing so therefore processing acti- 
vity would be illicit if another legal base could not be validly 
invoked. 


For example, the EDPB underlines that “any pressure or 
improper influence on the person (that may be manifested 
in different ways) preventing them from performing their will 
invalidates consent”. Unfair or deceptive design could also be 
seen as a desire of the processing manager to influence the 
person inappropriately. This influence should be read in light 
of the concept of balance of power, which the EDPB recalls 
can apply in all situations showing signs of constraint, deceit, 
intimidation, pressure or inability to exercise real choice. 


40 VSee French language guidelines on the CNIL 
website: https://www.cnil.fr/fr/reglement-europeen/lignes-directrices 
(viewed on 12.07.2018) 


41 See French language guidelines on the CNIL 
website: https://www.cnil.fr/fr/reglement-europeen/lignes-directrices 


(viewed on 12.07.2018) 
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Design and consent are tied, either positively, when design 
practices are aimed at improving the ability of individuals to 
make choices consciously or negatively, when they seek to 
deceive by abusive or misleading design practices. 


The creation of visual grammar or design patterns conceived 
to meet the exclusive interests of processing managers yet 
rehashed ad nauseam by all players, to the point that they 
become a sort of standard, can also lead to distorting consent. 
As underlined by the EDPB, users receive several consent 
requests on a daily basis to which they must respond with a 
click or by swiping their screens. This can lead to a certain 
fatigue: when too often encountered, the warning effect of 
the consent mechanisms diminishes. This results in a situation 
where no-one reads consent information any more. 


Indeed, the problem is not so much the creation of design 
standards as the multiplication of meaningless messages or 
calls to action for the individual. Such standardised accu- 
mulation of aberrant requests through the different services 
necessarily tires users out. 


Consent fatigue mentioned by some players is less an excuse 
than an additional reason to do better and innovate in an 
ethically unsatisfactory situation that can ultimately lead to a 
legally difficult situation”. 


Facilitating the exercise of rights 
by the design of pathways 


Individuals have a number of rights, which are also stren- 
gthened and supplemented by the GDPR: right of access, 
rectification, objection, erasure, portability and limiting of 
processing. 


The GDPR provides that organisations processing personal 
data must implement real practical solutions to enable the 
relevant persons (users, customers, employees, suppliers, etc.) 
to exercise these rights effectively. 


But the exercise of these rights is above all a matter of user 
pathway and context: information on the exercise of rights 
must be simple, practical, and present wherever it makes 
sense in the interfaces between the user and the service. 
It is necessary not only to recall at the right time that these 
rights exist but provide simple ways to exercise them. This 
simple and practical exercise of rights is in the interest of the 
organisation responsible for the data: the more organised this 
exercise, the less complex the deadlines. 


Beyond the amount of information to present to users, the 
very formatting of this information matters. The GDPR goes 
in this direction, making it enforceable. Can an information 
notice written in tiny characters also be regarded as “easily 
accessible”? Can a refusal to consent button with shades of 
colours and formatting making it almost invisible demonstrate 
valid “free and informed” consent? Could the fact that users 
answer positively through lassitude or by mistake to repeated 
data collection approval requests - bordering on harassment - 
be considered as a positive act by users? Is the imposition of 
an obstacle course for users to find where and how to claim 
their right of access and portability of data really compatible 
with the obligation to facilitate the exercise of rights? 


When a designer creates a system, their design choices ine- 
vitably influence the user. Such power is necessarily a res- 
ponsibility and qualifies its designers as “architects of choice” 
(Sunstein / Thaler)“*, a sort of conceptual counterpart of the 
concept of processing manager “who determines the pur- 
poses and ways of processing” (as defined in Article 4 of 
the GDPR). The architect of choice decides (intentionally or 
unintentionally) the social, technical and political environment 
in which individuals exercise their power to choose (or not to 
choose). The whole architecture of choice, whether intentio- 
nally designed to affect user behaviour or not, will affect how 
users interact with a system. 


The regulation of architectures of choice will perhaps be one 
of the most important regulatory fields of the digital society 
for the next 10 years, extending well beyond the issues of 
data protection and privacy. Regulators and legislators the- 
refore need to immediately build a rigorous analysis grid of 
architectures of choice and of their consequences on indivi- 
duals and on society, in an ethical and political approach that 
goes beyond both a purely legal and a simply instrumental 
approach to design. 


43 On the subject of rights, 
See the website of the CNIL: 
https://www.cnil.fr/fr 


42 VSee for example: 
https://www.beuc.eu/ blog / e-privacy-and-the-doorstep-salesmen / 
(viewed on 12.20.2018) 


44 On this subject, see: Cass Sunstein, Choosing not to choose, 2015. 
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BUILDING A SHARED AND OPEN 
REGULATION THROUGH NEW TOOLS 


Strengthening protection authorities 
with third-party, integrated and/or 
pooled competences to the benefit of 
regulation of architectures of choice 


If regulators want to continue to objectify the analysis of 
design and interface patterns, they need to develop profes- 
sional skills adapted to the rational and professional analysis 
of these issues. For some, these subjects seem less rigorous 
than law or technology, but design is not cursed with being 
a simple matter of instinct or taste: it covers practices that 
can be broken down and analysed. 


To add the missing side to the regulatory triangle, data pro- 
tection authorities in Europe need to recruit more skills to 
analyse interfaces, using ad hoc or more regular teams of 
designers and specialists in questions of psychology of indi- 
viduals. In some cases, it might be interesting to in-source 
these skills and expertise, it would also be possible to esta- 
blish cross-cutting laboratories, European data protection 
regulating inter-authorities or French regulating inter-au- 
thorities. This second scenario carries the risk of less honed 
applied expertise, but the probability that it will be more often 
shared beyond privacy protection. 


This growing competence of regulators is a necessary 
condition for reducing the imbalanced information between 
regulator and regulated, and therefore a condition of the 
effectiveness of public action in the digital age. 


Building a non-competitive 
and open source approach 
to best design practices 


A regulator like the CNIL acts on two levels: the suppor- 
tive level and the repressive level. If the controller could be 
led to consider design to decide on the non-compliance 
of certain practices, it may also help professionals create 
best practices. But that does not mean that creating these 
solutions is the job of the regulator: It must encourage it 
and not provide turnkey solutions. If that were the case, the 
regulator would exceed its role, would probably produce an 
inefficient outcome that would stifle opportunities for inno- 
vation and other players’ creativity, whose efforts would be 
considered superfluous by their peers and their professional 
interlocutors. 


Development and design professionals have codes, their 
own vocabulary and are complemented by a range of tools 
and design methods (guidelines, toolbox, design patterns, 
canvas) on which they usually rely. By their mass adoption, 
these design practices tend to homogenise forms of interac- 
tion and interfaces, which helps create grammars of inter- 
faces that form the basis of uses and interactions between 
humans and digital products. 


In addition to being subject to internal constraints (e.g. 
legal or marketing departments), the designers do not have 
enough tools available to formulate innovative responses 
to these new needs. They then fall back on their traditional 
tools and methods and leading UI and UX practices, which 
are not always appropriate (e.g. dark patterns, cookies and 
privacy policies that currently dominate, etc.). 


The CNIL could participate in the production of such tools, 
in an open format and under sharing licenses, to be conside- 
red as ways to make designers consider privacy. This could 
result in the production of analyses to support the design of 
interfaces that respect the privacy of users (acculturation to 
data protection subjects, issues to integrate into the design 
process, building blocks, major principles and rules, etc.) and 
concrete recommendations (“do” / “don't”, design patterns, 
typologies of transparency and loyalty mechanisms, etc.). 
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Parallel to the publication of this IP report, the CNIL plans to 
launch an initial version of this toolkit, as a way of opening 
the process, which should be progressively built and as a 
call to create a responsible design community in terms of 
data protection. 


Such tools could allow professionals to share their prac- 
tices and share their own approach to privacy issues to 
co-construct privacy design practice and bring together a 
design community on this topic. 


Focus on... 


Avenues for the regulation of 
architectures of choice 


The subject of architectures of choice has already 
been raised by the cross-cutting aspect of the 
work of the various regulators or public authorities. 
Research has already laid the groundwork: accor- 
ding to Cass Sunstein*’, the possible architectures 
of choice field can be summarised in a space ranging 
from default choice to active choice. In this space, 
many intermediate situations can occur, depending 
on the choice of tools, the selected rules or terms 
of implementation: simplified or advanced mode, 
general or customised, based on a firm rule or on 
nudges... Cass Sunstein lays the basic rules allowing 
an architect of choice to offer the “default” or active 
option to users, in any context, from health to privacy. 


For example, non-customisable default rules will be 
effective in a confused, technical context or misun- 
derstood context by users, where learning of the 
system has no major importance and where the user 
population is homogeneous. Active choice is a better 
solution when architects of choice are not neutral, the 
context is familiar and non-technical to users, when 
learning counts and individuals have a marked prefe- 
rence for being able to choose. Similarly, Alessandro 
Acquisti and his colleagues suggest the first “guide- 
lines for an ethical design of privacy nudges*®”. 


This work must now very directly provide input for 
regulatory and policy tools. It is an opportunity for 
regulators to further integrate reflection on privacy 
by design and privacy by default without excessively 
extending their actions. Indeed, actively regulating 
architectures of choice may be considered extremely 
paternalistic and coercive. This has led some mili- 
tants of soft paternalism or libertarian self-regulation 
to reject any idea of active regulation of these archi- 
tectures by public policies. Our position is rather to 
develop and strengthen tools allowing regulators to 
explore the preferences and choices of individuals, 
for example by asking to have access to more robust 
information on different choice pathways (the actual 
levels of opt-out or opt-in for example) or by promo- 
ting public debate and research on these subjects. 


45 Cass Sunstein, Choosing not to choose, 2015. 


46 Alessandro Acquisti et al, “Nudges for Privacy and Security: Understanding and Assisting Users’ 


Choices Online’, CM Computing Surveys (CSUR), 2017 
https://dl.acm.org/citation.cfm?id=3054926 
(viewed on 6/12/2018) 
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Encouraging retro-engineering 
of design practices 


The computer security industry regularly sets up Bug 
Bounties, programmes allowing developers to discover and 
correct bugs before the general public is informed, thus 
avoiding abuse. These programmes are launched by the 
companies themselves which offer rewards for developers 
who will enable them to identify - and therefore prevent or 
remedy - security vulnerabilities. In a design version, good 
market regulation is expected to reveal such initiatives direc- 
tly promoted by providers of digital services which would 
ensure their good practices by opening abusive and decep- 
tive design feedback channels. Players in innovation ecosys- 
tems could thus set up platform retro-design programmes, 
similar to what Fing is developing in its Retro Design of 
attention exploration’ and assign human science resear- 
chers and designers to dissect the process and highlight 
areas of improvement for digital platforms. 


Companies have become fond of hackathons, rapid prototy- 
ping or sprint design. Transposing the world of compliance 
and the implementation of design solutions to support the 
rights of users is therefore an opportunity to be grasped. 


Meanwhile, data and privacy regulators must also develop 
“regulation by reputational incentives” (sunshine regulation). 
Focusing on the implementation of transparency practices 
of players so that the general public may draw its own 
conclusions and may for example choose to leave a service 
showing malpractice: the reputation issue is crucial in eco- 
nomic models of platforms. This for example is what LINC 
has decided to do, by enforcing an article of the Loi pour 
une République Numérique (Act for a Digital Republic)*® 
giving the CNIL a mission to promote “the use of privacy 
protecting technologies, including data encryption techno- 
logies”, publishing a “mapping of tools and privacy protection 
practices” which references tools and services offering the 
ability, of different forms, of protecting the data of their uses 
embedded in their features or technology. 


Debating abusive or deceptive design practices in public 
could result in “market punishment” phenomena: an effective 
means of encouraging players to change their methods and 
inform individuals of how such practices are implemented. 
Such a mission is not necessarily the preserve of a single 
regulator: academics, advocacy groups, citizens and law- 
makers each have their role to play in this public debate. 
It is even a major issue of these additional pathways that 
control to better involve citizens in addition to traditional 
regulatory tools.°° 


47 See here: http://fing. 
org/?Pour-un-retrodesign-de-l-attention&lang=fr 


48 Act No. 2016-1321 of 7 October 2016 for 
a Digital Republic: 

https: // wwwlegifrance.gouv.fr/eli/ 
loi/2016/10/7/ ECFI1524250L / jo / Text 
(Viewed on 7/12/2018) 


49 https://linc.cnil.fr/une-cartographie- 
des-outils-et-pratiques-de-protection- 
de-la-vie-privee 


50 A prospect that agrees, for example, with 

the proposals by MP Paula Forteza, calling for 
“regulation by society. [.] To gain control of their 
online activities, users need tools, data, informa- 
tion: regulators must be able to provide them and 
become a resource platform’ 
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AS FUTURE REGULATION OPTIONS 


Funding of studies on the impacts of 
abusive or deceptive design 


Although scientific literature on abusive design practices is 
expanding, whether in the field of the economy of attention, 
behavioural economics, or psychology, etc. there is relatively 
little research work on the design of privacy. It is also recom- 
mended to encourage and support interdisciplinary university 
research in this field to better know, quantify and analyse 
concrete impacts of practices described in this document. 
Not only the regulator, but also the media and society as 
a whole might well seize the results of this work to better 
regulate, better inform and better respond to the demands 
of digital platforms. 


Supporting education in digital platforms 
and interfaces 


Digital literacy is an educational issue for young and old, in 
a world where all of our interactions tends to go through the 
digital and now natural interface vector (voice assistants, 
etc.). Each of these tools is developing its own grammar, 
its own language, with sometimes the desire to blur infor- 
mation to better influence individuals. The CNIL develops 
and manages the EducNum network, a collective born in 
2013 bringing together diverse stakeholders from the worlds 
of education, research, the digital economy, civil society, 
business foundations and institutions to carry and support 
actions to promote a true digital citizen culture. 


Pushing new initiatives in this way to educate in understan- 
ding platforms and interaction with interfaces will help limit 
the negative effects of abusive design attempts. The more 
vigilant people will be and able to recognise them, the fewer 
effects these manipulation attempts will have on internet 
users. Furthermore, as shown by our forward-looking sce- 
narios, one of the most intriguing questions about the future 
is the effect of these tools and practices on our brains and 
cognitive processes. However, it is not necessary to have 
a merely passive vision: learning is also changing our way 
of thinking, solving problems, responding to situations with 


a faster, instinctive and emotional cognitive mode (“system 
1” to use Kanheman’s distinction in his book Thinking fast 
and slow)". 


Often, in an entrenched republican tradition, digital education 
is thought to move towards “System 2”, the more analytical, 
logical and ... slower system. Individuals are explained, made 
to understand and guided towards new behaviours or new 
practices. But nothing prevents us from finding - with cau- 
tion - digital learning public policies more oriented towards 
system 1. For example, can we beef up reactance, that “psy- 
chological defence mechanism used by an individual who 
tries to keep their freedom of action when they believe it has 
been removed or threatened” (wikipedia) How to increase 
alertness, help people detect suspicious, sensitive or surpri- 
sing points? How can we think of ways to train citizens to 
react instinctively to defend their rights? 


In its summary report of the public debate that it hosted in 
2017 on the the ethical matters raised by algorithms and 
artificial intelligence”? , the CNIL had highlighted this great 
principle of vigilance: “The aim is to hold a regular, metho- 
dical and deliberative form of questioning regarding these 
moving objects”. Strengthening our individual and collective 
capacity for vigilance and reflexivity appears, in the digital 
society of the future, to be a worthy goal for public policy, in 
the public interest. 


Thinking about the subject also means applying the founding 
principles of the GDPR (informational self-determination and 
actual control by an informed individual, strengthening of 
rights, collective actions, etc.) and the 1978 Data Protection 
Act (in particular Article 1: “IT is there to serve every citizen. 
(..) Everyone has the right to decide and control the uses 
made of their personal data”. 


51 Daniel Kahneman, Thinking fast and slow, 2011. 


52 how can humans keep the upper hand? The ethical matters raised by algorithms and artificial 
intelligence, 
https://www.cnil.fr/sites/default/files/atoms/files/cnil_rapport_ai_gb_web.pdf 
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The foresight Committee 


The CNIL leads a committee of twenty-one experts with varied profiles and backgrounds to enrich projective reflections 
and contribute to discussions on digital ethics. Being more attentive and more open to the outside, working in partnership 
with the world of research and innovation are the objectives pursued by the CNIL through this Committee. 


Chaired by the President of the CNIL, Isabelle Falque-Pierrotin, 
the following people sit on the committee: 


OUTSIDE EXPERTS 


Pierre Bellanger, 
pioneer of free radio, entrepreneur and 
internet expert. 


Pierre-Jean Benghozi, 
member of the ARCEP college and professor 
at Ecole Polytechnique. 


Stefana Broadbent, 

psychologist, honorary professor of anthropo- 
logy at University College London where she 
teaches digital anthropology. 


Isabelle Bordry, 
entrepreneur, pioneer in the French digital 
media industry. 


Dominique Cardon, 

sociologist, associate professor at Sciences 
Po Medialab Paris, member of the editorial 
board of the journal Réseaux. 


Milad Doueihi, 
philosopher, historian of religions and holder of 
the Chair of digital humanism at the University 
of Paris-Sorbonne (Paris IV), co-holder of 

the Chair of the College des Bernardins on 
humanity facing the digital challenge. 


MEMBERS OF THE CNIL 


Joëlle Farchy, 

Professor in information science and communi- 
cations at the Paris | University and researcher at 
the economics centre of the Sorbonne. 


Célia Hodent, 
psychologist specialising in the application of 
he user experience in video game design. 


Claude Kirchner, 

nria research director 

Chairman of the operational evaluation com- 
mittee of legal and ethical risks (COERLE) 
nria, advisor to the President of INRIA. 


David Le Breton, 
Professor of Sociology and Anthropology at 
he University of Strasbourg. 


Titiou Lecoq, 
reelance journalist, blogger, novelist and 
essayist, specialist in web culture. 


Lionel Maurel, 
awyer, librarian and author of the S.l.L.ex blog, 
where he deciphers and analyses the changes 
in the law in the digital age. 


Cécile Méadel, 
sociologist, professor at Panthéon-Assas 
University, director of the Communications and 
multimedia master's degree. 

CARISM researcher, associate researcher 

at the Centre de Sociologie de l'Innovation 
(Mines-CNRS) 


Eric Pérès, 
Member of the economic, social and environ- 
mental council. 


Tristan Nitot, 

entrepreneur, author and speaker on the 
topic of digital freedoms, founded and chaired 
Mozilla Europe. 

He is the Advocacy VP at Qwant. 


Bruno Patino, 

journalist and specialist in digital media. 
Director of the Journalism School of Sciences 
Po. 


Antoinette Rouvroy, 

lawyer, FNRS researcher at the Centre de 
Recherche Information, Droit et Société 
(CRIDS) of Namur. 


Henri Verdier, 
Digital ambassador, Ministry of European and 
Foreign Affairs. 


Nicolas Vanbremeersch, 
entrepreneur, Chairman and founder of the 
Spintank agency and Le tank co-working 
space. 


Célia Zolynski, 

associate professor of private law at the Law 
School of the Sorbonne - Paris 1 Panthéon- 
Sorbonne. Member of the CERNA and 
qualified personality within the CSPLA. 


Valérie Peugeot, 
researcher at the laboratory of social sciences of 
Orange Labs. 


SHAPING CHOICES IN THE DIGITAL WORLD 
INNOVATION AND PROJECTIVE COLLECTION 


Innovation and Projective collection 


Within the Directorate of Technology and Innovation of the CNIL, the innovation, studies and 
projective team steers through research projects and explorations of emerging subjects linked 
to personal data and privacy. Its work is at the crossroads between innovation, technology, 
customs, society, regulation and ethics. 


The Innovation & Projective IP collection aims to present and share work and projective studies 
conducted by the CNIL. The aim is to contribute to interdisciplinary and open thinking in the 
Information Technology & Liberties field and fuel the debate on digital ethical topics. 


This issue is the 6th in this collection: 


IP 1 
Privacy between now and 2020. 
- Discussions by experts 


IP 2 
The body, the new connected object, from Quantified Self to mHealth: 
- the new territories of the world’s data layout 


IP 3 
The data, museums and boundaries of creation 
- Reading, listening, watching and playing in the era of customisation 


IP 4 
Foresight committee: Share! 
- Motivations, checks and balances to self-sharing in the digital society 


IP 5 
The platform of a city 
- Personal data at the heart of the smart city 


Find us also on the LINC editorial space (http://linc.cnil.fr). 
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Glossary 


ACRONYMS AND TERMS USED 


Affordance: sometimes translated as “potential”. Relationship 
between the properties of an object and the capacity of an agent 
determining how the object can potentially be used by the agent. 
Also referred to as intuitive use (or the intuitive character) of an 
object. 


Agency: a being's ability to act, their capacity to act on the world, 
things, beings, to transform or influence them. 


Call to action: marketing term referring to any device designed to 
generate or encourage the immediate action of an individual (e.g. 
a click). 


Cookie wall: technical device found on some websites preven- 
ting access to content until the person accepts the presence of 
cookies53 . 


Dark pattern: misleading user interface, carefully designed for a 
user to make choices without being aware of them, or which they 
do not want to do. 


EDPB: European Data Protection Board. 


Experience design (UX): design of the whole of the user 
pathway of a tool or service, beyond interfaces. 


FOMO: Fear Of Missing Out 
GAFA / GAFAM: Google Amazon Facebook Apple (Microsoft). 


GDPR: General Data Protection Regulation 2016/679 EU 
Regulation of the European Parliament and of the Council of 27 
April 2016 on the protection of individuals with regard to the pro- 
cessing of personal data and on the free movement of such data, 
revoking Directive 95/46/EC). 


Human-machine interactions (or interfaces) (HMI): Human- 
machine interactions (HMI) define the means and tools imple- 
mented so that a human can control and communicate with a 
machine. 


Interaction design (IxD): design of the behaviour of an interface 
to give meaning to the interactions of a user with a system and 
enable them to achieve their objectives. 


Interface Design (UI): design of visual or sensory elements of 
the interface to allow the user to read and to be guided in their 
interactions with them. 


Legal basis: in the GDPR, Article 6 lists six legal bases (consent, 
performance of a contract, legal obligation, protection of the vital 
interests of the person, ...) on which data processing can be based 
to be lawful. These are the “legal bases” of this processing. 


Natural User Interfaces (NUI): common term used for 
human-machine interfaces to refer to a user interface that is and 
remains invisible as the user performs various interactions. The 
word natural is used because most computer interfaces use artifi- 
cial control devices that require an apprenticeship. The reference 
“natural” is subject to caution in this expression. 


Nudge: technique to encourage individuals or a target population 
to change their behaviour or to make certain choices without 
being under duress or obligation and does not involve any 
sanctions. 


Privacy by Design: protection of privacy from the design stage. 
The concept is taken up in article 25 of the GDPR. 


Privacy policy: contract that describes how a company retains, 
processes, publishes and removes data transmitted by its 
customers. 


Reactance: psychological defence mechanism used by an indivi- 
dual who tries to keep their freedom of action when they believe it 
has been removed or threatened. 


53 See documents concerning the revision of Directive 002/58 / EC or ePrivacy Directive. 


https://edpb.europa.eu/sites/edpb/files/files/file 1 /edpb_statement_on_eprivacy_en.pdf or https://ec.europa.eu/transparency/regdoc/rep/10102/2017/EN/SWD-2017-3-F1-EN-MAIN-PART-3.PDF, 


(viewed on 13/12/2018) 
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